Skip to content

fix(agents): harden format guard review findings - #2995

Merged
stranske merged 3 commits into
mainfrom
agent/sync-review-fix-5800
Aug 9, 2026
Merged

fix(agents): harden format guard review findings#2995
stranske merged 3 commits into
mainfrom
agent/sync-review-fix-5800

Conversation

@stranske

@stranske stranske commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • defer API-client setup until a non-exempt, non-held issue needs the optimizer path
  • re-fetch and revalidate before clearing agents:formatted; skip closed issues before any routing side effect
  • accept lower-camel task identifiers such as calculateDiscount

Validation

  • python -m pytest -q tests/scripts/test_issue_format.py tests/workflows/test_agents_issue_optimizer_format_trigger.py (42 passed)
  • python scripts/check_template_drift.py --allowlist config/template-drift-allowlist.txt (0 unallowlisted drift)
  • root/template parity checks plus git diff --check

Related to source-owned review debt on stranske/Trend_Model_Project#5800.

Summary by CodeRabbit

  • Bug Fixes

    • Improved recognition of concrete task targets, including lowerCamelCase function names.
    • Prevented unnecessary processing for exempt, held, or closed issues.
    • Improved handling of stale formatting results when issues are updated or closed.
    • Ensured optimizer actions are only triggered for open issues.
  • Tests

    • Added coverage for function-oriented task descriptions and closed-issue workflow scenarios.

Copilot AI lite review requested due to automatic review settings August 8, 2026 23:30
@cursor

cursor Bot commented Aug 8, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 27 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: cae9069d-1fef-4015-b307-8cf1e0bcd72d

📥 Commits

Reviewing files that changed from the base of the PR and between 4b077cd and 80def37.

📒 Files selected for processing (5)
  • .github/scripts/issue_format.py
  • .github/workflows/agents-issue-format-guard.yml
  • templates/consumer-repo/.github/scripts/issue_format.py
  • templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml
  • tests/scripts/test_issue_format.py
📝 Walkthrough

Walkthrough

The validator now detects more concrete task targets. The format-guard workflow delays API setup, revalidates live issue state before label changes, and dispatches the optimizer only for open issues. Tests cover these paths.

Changes

Issue format guard

Layer / File(s) Summary
Concrete task target validation
.github/scripts/issue_format.py, templates/consumer-repo/.github/scripts/issue_format.py, tests/scripts/test_issue_format.py
The validator accepts lowerCamelCase symbols and unquoted identifier-like targets. Tests cover calculateDiscount.
Issue resolution and action gating
.github/workflows/agents-issue-format-guard.yml, templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml
The workflow exports issue state, conditionally sets up the API client, and requires open issues before optimizer routing.
Live stale-format revalidation
.github/workflows/agents-issue-format-guard.yml, templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml, tests/workflows/test_agents_issue_optimizer_format_trigger.py
The workflow re-fetches issues before removing agents:formatted, preserves the label for closed or valid issues, and handles validator failures explicitly. Tests cover gating and revalidation.
Worker attempt metadata
langsmith-fleet-worker-attempt.json
The worker metadata records a new emission timestamp and PR number.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Workflow
  participant IssueResolver
  participant GitHubAPI
  participant issue_format_py
  participant Optimizer
  Workflow->>IssueResolver: Resolve issue and export state
  Workflow->>GitHubAPI: Fetch live issue
  GitHubAPI-->>Workflow: Return state and body
  Workflow->>issue_format_py: Validate issue body
  issue_format_py-->>Workflow: Return validation result
  Workflow->>GitHubAPI: Preserve or remove agents:formatted
  Workflow->>Optimizer: Dispatch only for open valid issues
Loading

Possibly related issues

Possibly related PRs

Suggested labels: verify:compare

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: strengthening format-guard behavior based on review findings.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/sync-review-fix-5800

Comment @coderabbitai help to get the list of available commands.

@stranske
stranske temporarily deployed to agent-high-privilege August 8, 2026 23:30 — with GitHub Actions Inactive
@agents-workflows-bot

Copy link
Copy Markdown
Contributor

Workflow source needed

PR #2995 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely.

Please do one of:

  • Add <!-- meta:issue:123 --> or a normal Closes #123 / Related to #123 line.
  • Check one Workflow Source option in the PR body.
  • Add a hidden marker such as <!-- workflow-source:local_request -->, <!-- workflow-source:manual_remote -->, <!-- workflow-source:review_followup -->, <!-- workflow-source:sync_campaign -->, or <!-- workflow-source:dependabot -->.
  • Add a workflow source label such as workflow:source-direct-pr, workflow:source-local-request, workflow:source-review-followup, workflow:source-sync, or workflow:no-automation.

Once a valid source is present, this warning will not be reposted.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Hardens the “issue format guard” workflow and shared issue-format validator to reduce unnecessary side effects (especially for exempt/held/closed issues) and improve recognition of concrete task targets in issue bodies across both this repo and the consumer template.

Changes:

  • Defer API client setup until it’s actually needed (non-exempt and non-held) and add issue state (OPEN/CLOSED) to resolved outputs.
  • Re-fetch and re-validate the live issue body before clearing agents:formatted, and skip optimizer routing (and label side effects) for closed issues.
  • Expand “concrete target” detection to include lowerCamelCase identifiers and unquoted symbol tokens; update tests accordingly.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
.github/workflows/agents-issue-format-guard.yml Defers API client setup, adds state output, and revalidates live issue before label/routing side effects; skips closed issues.
templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml Mirrors the same guard hardening changes for consumer repos.
.github/scripts/issue_format.py Accepts lowerCamelCase identifiers and recognizes concrete unquoted symbol tokens in task items.
templates/consumer-repo/.github/scripts/issue_format.py Mirrors the same validator logic changes for consumer repos.
tests/workflows/test_agents_issue_optimizer_format_trigger.py Adds assertions covering new guard ordering/conditions and closed-issue skipping behavior.
tests/scripts/test_issue_format.py Extends validator coverage to ensure lowerCamelCase identifiers are accepted as concrete task targets.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78b4d712c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/issue_format.py
@stranske-keepalive

stranske-keepalive Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Automated Status Summary

Head SHA: a6ad98f
Latest Runs: ⏳ pending — Gate
Required contexts: summary
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending

Coverage Overview

  • Coverage history entries: 1

Coverage Trend

Metric Value
Current 76.21%
Baseline 85.00%
Delta -8.79%
Minimum 70.00%
Status ✅ Pass

Top Coverage Hotspots (lowest coverage)

File Coverage Missing
scripts/issue_dedup_smoke.py 0.0% 4
scripts/runner_lib/__main__.py 0.0% 3
scripts/validate_template_sync.py 0.0% 81
scripts/langchain/topic_splitter.py 19.1% 57
tools/codex_log_analyzer.py 19.6% 140
scripts/repo_review_round2_runner.py 25.1% 344
scripts/prune_agent_stubs.py 39.7% 26
scripts/repo_review_round1_runner.py 40.7% 133
tools/ensure_workflow_timeout_variables.py 42.1% 74
scripts/sync_label_docs.py 42.9% 64
tools/discover_model_catalog.py 44.8% 55
scripts/repo_review_backlog_scan.py 45.3% 116
scripts/repo_review_body_writer.py 46.5% 86
tools/codex_session_analyzer.py 47.9% 59
scripts/create_verifier_labels.py 48.3% 58

Low Coverage Files (<50.0%)

File Coverage Missing
scripts/issue_dedup_smoke.py 0.0% 4
scripts/runner_lib/__main__.py 0.0% 3
scripts/validate_template_sync.py 0.0% 81
scripts/langchain/topic_splitter.py 19.1% 57
tools/codex_log_analyzer.py 19.6% 140
scripts/repo_review_round2_runner.py 25.1% 344
scripts/prune_agent_stubs.py 39.7% 26
scripts/repo_review_round1_runner.py 40.7% 133
tools/ensure_workflow_timeout_variables.py 42.1% 74
scripts/sync_label_docs.py 42.9% 64
tools/discover_model_catalog.py 44.8% 55
scripts/repo_review_backlog_scan.py 45.3% 116
scripts/repo_review_body_writer.py 46.5% 86
tools/codex_session_analyzer.py 47.9% 59
scripts/create_verifier_labels.py 48.3% 58

Updated automatically; will refresh on subsequent CI/Docker completions.


Keepalive checklist

Scope

No scope information available

Tasks

  • No tasks defined

Acceptance criteria

  • No acceptance criteria defined

@stranske

stranske commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Runner dispatch state for autofix on PR #2995. Do not edit.

@github-actions github-actions Bot added the autofix Opt-in automated formatting & lint remediation label Aug 8, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Runner dispatch state for codex on PR #2995. Do not edit.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Autofix updated these files:

  • tests/workflows/test_agents_issue_optimizer_format_trigger.py

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/agents-issue-format-guard.yml:
- Around line 68-75: Move Setup API client in
.github/workflows/agents-issue-format-guard.yml after validation and require the
issue to be open, non-exempt, non-held, and steps.validate.outputs.rc == '1'.
Apply the same workflow change in
templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml. Update
tests/workflows/test_agents_issue_optimizer_format_trigger.py to assert
validation precedes setup and setup requires state == 'OPEN' and rc == '1'.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: bb66df3b-e5c1-46ac-a378-c5787826d74f

📥 Commits

Reviewing files that changed from the base of the PR and between c251301 and 2f8df22.

📒 Files selected for processing (6)
  • .github/scripts/issue_format.py
  • .github/workflows/agents-issue-format-guard.yml
  • templates/consumer-repo/.github/scripts/issue_format.py
  • templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml
  • tests/scripts/test_issue_format.py
  • tests/workflows/test_agents_issue_optimizer_format_trigger.py

Comment thread .github/workflows/agents-issue-format-guard.yml
@stranske

stranske commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Closer recovery (cursor) — Codex P2 camelCase + CI

Head:

Autofix already landed the Black reformat () and a langsmith worker bump (). This closer commit rebases on those and adds:

  1. Codex P2: unquoted concrete-target matching requires unambiguous lowerCamelCase (). Brand/prose capitals (, , usage: openai [-h] [-v] [-b API_BASE] [-k API_KEY] [-p PROXY [PROXY ...]]
    [-o ORGANIZATION] [-t {openai,azure}]
    [--api-version API_VERSION] [--azure-endpoint AZURE_ENDPOINT]
    [--azure-ad-token AZURE_AD_TOKEN] [-V]
    {api,tools,migrate,grit} ...

positional arguments:
{api,tools,migrate,grit}
api Direct API calls
tools Client side tools for convenience

options:
-h, --help show this help message and exit
-v, --verbose Set verbosity.
-b API_BASE, --api-base API_BASE
What API base url to use.
-k API_KEY, --api-key API_KEY
What API key to use.
-p PROXY [PROXY ...], --proxy PROXY [PROXY ...]
What proxy to use.
-o ORGANIZATION, --organization ORGANIZATION
Which organization to run as (will use your default
organization if not specified)
-t {openai,azure}, --api-type {openai,azure}
The backend API to call, must be openai or azure
--api-version API_VERSION
The Azure API version, e.g.
'https://learn.microsoft.com/en-us/azure/ai-
services/openai/reference#rest-api-versioning'
--azure-endpoint AZURE_ENDPOINT
The Azure endpoint, e.g.
'https://endpoint.openai.azure.com'
--azure-ad-token AZURE_AD_TOKEN
A token from Azure Active Directory,
https://www.microsoft.com/en-
us/security/business/identity-access/microsoft-entra-
id
-V, --version show program's version number and exit) no longer satisfy. Backticked PascalCase () unchanged. Mirrored in consumer template.
2. Regression tests for GitHub/JavaScript vague tasks.

Validation

no tests ran in 0.00s → 44 passed

  • on touched files → clean
  • template drift allowlist check → 0 unallowlisted

Review thread resolved on this head. Awaiting fresh CI + 7-minute post-push review window before merge (mandatory post-push invariant).

@stranske

stranske commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Closer recovery (cursor) — Codex P2 camelCase + CI

Head: abad9ab2298c157730b70c547e8406dfbe98057e

Autofix already landed the Black reformat (2f8df222) and a langsmith worker bump (4b077cde). This closer commit rebases on those and adds:

  1. Codex P2: unquoted concrete-target matching requires unambiguous lowerCamelCase (calculateDiscount). Brand/prose capitals (GitHub, JavaScript, OpenAI) no longer satisfy. Backticked PascalCase (IssueFormatter) unchanged. Mirrored in consumer template.
  2. Regression tests for GitHub/JavaScript vague tasks.

Validation

  • pytest -q tests/scripts/test_issue_format.py tests/workflows/test_agents_issue_optimizer_format_trigger.py44 passed
  • black --check --fast on touched files → clean
  • template drift allowlist check → 0 unallowlisted

Review threads resolved on this head. Awaiting fresh CI + 7-minute post-push review window before merge (mandatory post-push invariant).

codex-automation and others added 3 commits August 8, 2026 19:24
Restrict unquoted concrete-target matching to lowerCamelCase so brand
prose like GitHub/JavaScript cannot falsely satisfy the format guard,
and reformat the optimizer trigger tests for CI black.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stranske
stranske force-pushed the agent/sync-review-fix-5800 branch from abad9ab to 80def37 Compare August 9, 2026 00:26
@stranske
stranske temporarily deployed to agent-high-privilege August 9, 2026 00:26 — with GitHub Actions Inactive
@stranske

stranske commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Closer recovery: rebased the exact PR head onto current main and force-pushed 80def37. The only conflict was generated langsmith-fleet-worker-attempt attribution; preserved main's current PR #2997 attribution rather than overwriting it. Validation: 44 focused tests passed; API-wrapper guard, template completeness/sync, and diff check passed. CI/review window now restarts from the new head before any merge.

@stranske
stranske enabled auto-merge (squash) August 9, 2026 00:36
@stranske
stranske merged commit b9c8ac4 into main Aug 9, 2026
50 checks passed
@stranske
stranske deleted the agent/sync-review-fix-5800 branch August 9, 2026 00:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autofix:patch autofix Opt-in automated formatting & lint remediation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants